1. Field of the Invention
The present invention relates to a real time video processing apparatus which receives a video signal transmitted at first timing, stores a part or the whole of the video signal in a storage medium, generates a video image on the basis of information read out from the storage medium, and outputs a video signal at second timing, and relates to such a video processing method and program.
2. Description of the Background
In the case of the so-called multipoint video conference in which communication is performed among a plurality of terminals by using microphone voices and camera video images, the load is heavy in the aspect of communication and terminal processing if simply the terminals are connected in a full mesh form. In general, therefore, a technique of providing an MCU (Multipoint Control Unit) is used. The MCU is a kind of a server, and has a function of connecting with terminals, receiving voice and video images from the terminals, composing them, and transmitting resultant composite video images and voices to the terminals. Owing to this MCU, it becomes possible for the terminals to obtain voices and video images from all participants by only communicating with the MCU and receiving a composite voice and a composite video image, resulting in a high efficiency in the aspect of communication and terminal processing. In this way, the MCU plays an important role in the multipoint video conference. Here, the real time video processing technique is utilized.
As another application of the real time video processing technique, there is a screen splitting unit in a surveillance camera. In general, surveillance cameras are installed in a building or the like. If separate monitors are used for respective surveillance cameras, the equipment becomes large-scaled, resulting in lowered convenience. In a typically used technique, therefore, a screen splitting unit is used, and a plurality of camera video images are composed to generate one video signal. The one video signal is ascertained using a single monitor or recorded using a single video recorder.
In this way, the video processing technique is already used in the fields of video conference and surveillance camera frequently. A large number of inventions concerning the video processing technique have been proposed. As for, for example, video conference, JP-As 9-270954(KOKAI), 10-164566(KOKAI), 11-187372(KOKAI) and 11-88854(KOKAI) can be mentioned. As for the surveillance camera and the like, JP-As 10-164566(KOKAI) and 11-234654(KOKAI) can be mentioned.
In a general form of such a video processing technique, basically predetermined video processing is performed on input video images while using a frame buffer such as a RAM and an output video image is generated.
Video frames are successively written into a RAM, and the video frames are successively read out. If there is time to spare, no problems are caused by performing them alternately. When both writing and reading are performed successively without interruption in the real time processing as described above and the writing rate is different from the reading rate, there is a possibility that reading will be performed with writing unfinished unless any countermeasure is taken. For example, a portion in which new update has been reflected and an old portion in which new update has not been reflected are mixedly present in a video frame read out. In this case, an immediately preceding video frame is partially mixed into a video image read out, and the video image is ruined. FIG. 13A shows its situation.
In order to prevent access contention between the writing side and the reading side, a technique called double buffer processing is used in general. Finely, the double buffer processing has a plurality of meanings. Here, however, the double buffer processing means a method of preparing two frame buffers on the RAM and thereby preventing the above-described access contention. For example, when the writing side writes frames, the frames are written alternately into the two buffers. On the other hand, the reading side reads a frame from a buffer from which writing is not being performed. In other words, operation is performed so as to alternately interchange a buffer to write in and a buffer to read out. By doing so, the buffer to read out is always completed in writing. As a result, a perfect state is ensured and video images are prevented from being ruined.
Basically, in the conventional double buffer processing, a method of causing one of writing and reading to access two buffers alternately and causing the other of writing and reading to always access a buffer which is not being accessed as described above is typical. In this method, however, a problem occurs when frames in the input video image and the output video image are not synchronized. For example, it is supposed that frames in the input video image and the output video image are not synchronized in the video processing apparatus exemplified earlier.
It is supposed that there is a video processing apparatus having a configuration in which an input video image is written into a RAM, read out, upscaled by a factor of two, and output. A double buffer is formed on the RAM as shown in a left-hand drawing in FIG. 13B. Writing is performed on the whole video frame. However, reading is performed only on a range of a shaded portion shown in the left-hand drawing in FIG. 13B. A portion obtained by upscaling the shaded portion to twice is disposed in an output video frame as shown in a right-hand drawing in FIG. 13B. When a reader is performing processing on a portion indicated by a solid line arrow Y21 shown in the right-hand drawing in FIG. 13B, a writer is performing processing on a portion indicated by a solid line arrow Y23 shown in the left-hand drawing in FIG. 13B. At this time, the solid line arrow Y23 of the writer side is in a buffer 1. In the conventional double buffer processing, therefore, the reader reads out from a buffer 2 as shown in FIG. 13B. If the processing on the output video image moves to a dotted line arrow Y22 in a right-hand drawing and reading from the buffer 2 is completed, then the writer proceeds by the same quantity and moves to a dotted line arrow Y24 in a left-hand drawing. This location has exceeded the lower end of a read region, and consequently it is meant that the writer side outruns the reader side on the way. Before the time when outrunning occurs, the video image read out becomes an immediately preceding frame as compared with that obtained after the time. Thus, video images in different frames are mixedly present, resulting in a ruined video image.
It is now supposed that an input video image is downscaled to ½, written into a frame buffer, read out, and output intact. This time, the frame buffers are downscaled to ½ as shown in a left-hand drawing in FIG. 13C. Since writing is performed in synchronism with the input video image, the rate at which the write access moves downward is also reduced to ½. In the same way as the foregoing description, the reader reads only a shaded portion, and disposes it in an output video image. If the writer performs processing on a portion indicated by a solid line arrow Y27 shown in the left-hand drawing in FIG. 13C when the reader performs processing on a portion indicated by a solid line arrow Y25, then in the conventional method data is read out from the buffer 1 which is not being accessed by the writer as illustrated. When an arrow shown in a right-hand drawing in FIG. 13C has arrived at a terminating dotted line arrow Y26, however, the access location of the writer moves to a portion indicated by a dotted line arrow Y28 as shown in the left-hand drawing in FIG. 13C at a half rate, and a video image of a shaded portion in the buffer 2 has been completely updated. Eventually, therefore, a newer frame would be obtained without ruining if reading was performed from the buffer 2.
Thus, in the conventional method, access contention cannot be excluded completely when, for example, frame synchronization is not attained. In addition, the latest frame is not read out in some cases. Since a plurality of video images are input and output in, for example, a video composing apparatus, it becomes difficult to attain frame synchronization between them for various reasons in some cases. In addition, since bi-directional communication is demanded in video conference and the like, the real time property of video images becomes very important and it poses a problem that the latest frame is not read out.
As a related technique, JP-A 5-212913(KOKAI) can be mentioned. A technique of determining which buffer to access according to a certain decision method is proposed therein. However, this technique is different from the above-described problem in that it is specialized in printers.